AsyncAPI 3.x: an E2E test, NCS driven through Kafka - #1760
Draft
LautaroPetaccio wants to merge 18 commits into
Draft
LautaroPetaccio wants to merge 18 commits into
LautaroPetaccio wants to merge 18 commits into
Conversation
LautaroPetaccio
force-pushed
the
feature/asyncapi-e2e-kafka
branch
from
September 14, 2026 11:59
4b4a66a to
64b5a5a
Compare
LautaroPetaccio
force-pushed
the
feature/asyncapi-e2e-kafka
branch
from
September 14, 2026 22:19
64b5a5a to
e69f386
Compare
LautaroPetaccio
force-pushed
the
feature/asyncapi-e2e-kafka
branch
from
September 15, 2026 23:46
e69f386 to
8e17ab8
Compare
LautaroPetaccio
force-pushed
the
feature/asyncapi-e2e-kafka
branch
from
September 16, 2026 21:59
8e17ab8 to
fd3a8b3
Compare
LautaroPetaccio
force-pushed
the
feature/asyncapi-e2e-kafka
branch
from
September 16, 2026 22:01
fd3a8b3 to
04946ba
Compare
LautaroPetaccio
added this pull request to stack #1712
September 20, 2026 22:02
LautaroPetaccio
force-pushed
the
feature/asyncapi-e2e-kafka
branch
from
September 20, 2026 22:05
4c564f5 to
df95194
Compare
ProblemType.ASYNCAPI, experimental like the other recent types. Main infers it from SutInfoDto.asyncApiProblem, picks its algorithm key, and stops with a clear message where the module will go: the sampler and fitness that complete it are the next changes. The one decision here is EMConfig.usesDriver(). Every other type reaches the driver only in white-box mode, or in black-box experiments. An AsyncAPI service has no universal wire to point at, so the driver holds the broker connection even when the service is a black box. The places in Main and Statistics that asked "is there a driver" now ask the config, instead of each spelling the rule out.
AsyncApiSampler asks the driver where the document is, reads it, builds one action per publishable message, and samples sequences of them. The first individuals it hands out publish one message each, one per operation, so every operation is tried before the search starts combining them. One sampler for both modes. Where REST and GraphQL fetch the schema themselves in black-box mode, here the document comes through the driver either way, as the driver is what holds the connection to the broker. Seeded test cases are not supported yet, and say so.
Review: what the parser skips was reported to the user and kept for the final report, but nothing checked it; nor that the single-message individuals return after a reset, nor the two other ways start-up can fail. Four tests, and the helper that builds the injector is shared.
Four TODOs, where someone will next look: no AsyncAPI-specific smart strategy yet, message examples parsed but unused, no seeding format, and a method shared verbatim with the REST sampler.
RemoteController.executeNewAsyncApiActionAndGetReply PUTs the action to the driver and reads back an AsyncApiReplyDto, as its RPC counterpart does. It has a default that throws, so the controllers that never publish need not say so one by one. EMConfig gains asyncApiReplyTimeoutMs, experimental, and a constraint: there is no test writer for AsyncAPI yet, so a run must say --createTests false rather than search for an hour and fail at the end. The two tests that already parsed --problemType ASYNCAPI say so now. Two experimental fault categories: a promised reply that never arrives, and a reply matching none of the messages the contract declares.
AsyncApiBlackBoxFitness publishes each message of a test through the driver and turns what comes back into targets: for every operation, what publishing to it was seen to do, and, when a reply came back, which of the messages the contract declares for the reply it was. A contract listing a result and an error thus gives the search two things to reach, which is the AsyncAPI analogue of REST's (status x endpoint). AsyncApiReplyClassifier is what recognises a reply: a structural match against each declared payload schema, reading the parts of JSON Schema that tell one message from another and giving the benefit of the doubt on the rest. The most specific match wins. AsyncApiModule binds it all, with the driver bound unconditionally, and Main uses it in place of the message it showed until now. One test runs a whole MIO search against a stand-in for the NCS service and sees both declared replies of an operation covered.
Review: with room for exactly two messages, a one-message test was mutated by removal, leaving a test that publishes nothing; and a test could never grow to the maximum the user allowed, only to one less. Both bounds are now stated as such, and AsyncApiStructureMutatorTest holds them. Seeding test cases is refused at start-up, like writing them, instead of failing inside the sampler. The whole-search suite is named after the class it exercises, AsyncApiModule, and the three suites that build the injector share how they do it.
Review: the ':' joining a target id and the '-' inside a correlation id were spelled out where used; they are constants now, and the two kinds of target id are built in one place each. The remote controller named its queryFromDatabase parameter four times over, once in the new call and three in the ones it mirrors; it is a constant now. The fake driver's one field comes before its companion.
…aults
Review findings, in order of what they cost a run.
The address a message is published to ignored protocol bindings. A Kafka
channel usually names no address and carries its topic in its binding --
microcks.yaml, already a test resource here, is exactly that shape -- so
messages went to the channel's name and reached nobody. The parser's
effectiveAddress() exists for this and was never called; when a channel
names no address at all, the binding's topic is now taken as the one
destination the document actually gives. An address left holding
{placeholders} is warned about rather than published to silently.
Deciding whether a number is an integer asked Jackson for a BigDecimal,
which throws for a value that overflows a double. Nothing between the
classifier and the search loop catches it, so one odd reply ended the run.
Three ways a valid reply was reported as a fault: a body that is empty or
not JSON, a const or enum written 1.0 against a reply carrying 1, and a
type or combinator this cannot read. A reply with nothing to match against
is now no finding at all, numbers compare by value, and what cannot be read
rejects nothing. A $ref that points into a schema is followed instead of
matching everything, which had quietly disabled the undeclared-reply oracle
for any operation written that way, and specificity counts through the
combinators so the more specific message wins.
Faults ignored isEnabledFaultCategory, so two experimental oracles fired on
a default run and --disabledOracleCodes did nothing. They are recorded on
the action result now, which is where the reports count faults from; before,
a run covered fault targets while every report said zero.
AsyncApiCallResult was the only action result not overriding matchedType.
An inferred problem type no longer dies on the test-generation constraint:
when the driver reports an AsyncAPI service, createTests is turned off with
a warning instead of throwing after the SUT has already been started.
The class is AsyncApiFitness: it is bound as the only fitness and reads
white-box coverage, so BlackBox in its name said something untrue.
The new remote-controller call was a copy of the RPC one; both now share the body that PUTs an action and reads the reply back. The structure mutator is a third copy of logic REST and RPC already share, marked with a TODO the way the GraphQL one marks the same debt. The reply-timeout description is published verbatim into options.md, where every neighbour is one clause; the reasoning behind the option belongs in the pull request, not in the user's terminal.
The guards that keep a valid reply from being reported as a fault had no tests, which is why they could be wrong. Now covered: a reply with no body, an operation whose reply declares no message, a number too large to be a decimal, a const compared by value, a reference into a schema, and the most specific match found through a combinator. Also covered: the topic a protocol binding names, the configured reply timeout, and that faults stay quiet until experimental oracles are asked for -- each of which would otherwise pass with the code deleted. The structure mutator's tracking is exercised with a real specification rather than null, so the add/remove bookkeeping the archive reads is checked. The module test asserts the bindings Main resolves, since this module deliberately inherits none. The sampler suite resets the static gene cache its siblings all reset.
The reply DTO no longer uses primitives, so a field the driver never set arrives as null rather than as a default. Kotlin maps those to platform types, so reading them as before would still compile and throw at run time. Each is now read deliberately. Whether the message was published has no safe default, so an absent answer is treated as a failure and said so in the error message, apart from a driver that answered false. Whether a reply was expected or arrived is read as not having happened when unset, which is what a fire-and-forget driver leaves blank. Correlation is the one that changes behaviour: a driver that does not track it at all reported false, which reads as the service having failed to echo the id back. That is a claim about the SUT the driver never made, so it is now recorded only when the driver actually looked.
Turning the experimental oracles off used to cost coverage rather than only reporting. A reply matching none of the declared messages registered nothing beyond the gated fault call, so it was indistinguishable from a recognised one and could be dropped when the solution was minimised. It now covers a target of its own, registered before the fault and independent of whether it is enabled, which is how a 500 is already handled for REST. Also in this round: - the correlation id prefix no longer comes from the seeded generator, which made a run repeated under the same seed reuse the very ids it exists to tell apart - an outcome that cannot reach target handling now says so by throwing rather than by a comment, and the branch stays exhaustive so that a newly added outcome fails to compile instead of falling through - why there is no separate black-box fitness, and why the reply classifier is written by hand rather than delegated to the validator already on the classpath, which reads draft-04 and so does not know const
A document's message examples are what its author knows the service accepts. A service that silently drops what it does not recognise may never answer a sampled payload, so with --probAsyncApiExamples the first example of a message is offered as a whole value beside the schema-derived genes, at that probability. Off by default, like every new feature. It reuses what REST already does with a schema's 'example': the example is put on the message's own copy of the payload schema, and the gene builder turns it into the same choice it offers REST. Only the first example is used: the OpenAPI parser the genes go through reads the schemas as 3.0, which drops the plural 'examples'. Scalars are left alone, as the builder would complain about 'example' on them. A headers example loses the field the correlation id is stamped into, as the headers gene did before it.
The first cut wrote one example into the message's copy of its payload schema and let the gene builder find it there. That path crosses the OpenAPI parser, which keeps only the first of several, drops the name an example may carry, and warns about an example on a scalar. So only one example was offered, never by name, and never for a scalar payload. The gene builder already takes examples the other way, as a list of values with optional names, for a REST parameter or request body. Its entry point for a bare schema simply did not expose that parameter. It does now, with an empty default so that no other caller changes, and the AsyncAPI builder passes every example through it: all of them are offered, a name survives to where the sampler's named-example pass can read it, and a scalar payload gets the same choice an object does. One trap came with it. The builder caches what it builds by schema text alone, so two messages sharing a schema but declaring different examples would have been handed the same gene. A call that brings examples now neither reads nor writes that cache.
The first search against a real broker. The SUT is the NCS case study re-expressed as a Kafka service: six request topics answered on six reply topics with a result or an error, as ncs-kafka.yaml describes. It speaks only Kafka; there is no HTTP endpoint. Its routines are ported from EMB's NCS and cited as such. The driver owns the broker, a testcontainers apache/kafka, and is the reference implementation of SutController.executeAsyncApiAction: publish with the correlation id where the document says it goes, wait for the reply that carries it back, report and do not judge. kafka-clients and testcontainers-kafka are used only here, in the test tree, and are managed in the root pom like everything else. The test asserts that every operation answers and that expint and gammq each reach both their declared replies. Those two are the operations whose rejected inputs the schema does not rule out; bessj's and remainder's bounds are in the schema, so their errors need invalid data, which the search does not publish yet. 662 targets in 26 seconds, on two runs.
Neither side of the experimental oracle gate was exercised against a live broker. A second search now runs with the oracles on and a reply deadline no round trip can meet, so every message goes unanswered and the fault is reported, while the existing search asserts that a service which answers correctly has nothing reported against it. Nothing in the service is broken on purpose for this. The driver ignores replies whose correlation id does not match the request it is waiting on, so the late answers piling up on the reply topics cannot be taken for fresh ones.
arcuri82
force-pushed
the
feature/asyncapi-e2e-kafka
branch
from
September 22, 2026 11:05
df95194 to
5609333
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The SUT — NCS over Kafka
The NCS case study re-expressed as a message-driven service: six request topics (
ncs.<op>.request), each answered on a reply topic with a result message or, for the inputs the REST version answers with a 400, an error message — exactly whatncs-kafka.yaml(already in core's test resources) describes. The routines are ported from EMB'sorg.restncs.impand cited at the top of each file; the rejection rules mirrorNcsRest. A Spring Boot application with no web layer: it speaks only Kafka, which is the point.One deliberate deviation: a result that is
NaNor infinite is answered as an error. JSON has no such numbers, and a string where the contract promises a number would be a false fault.The driver — the reference
executeAsyncApiActionNcsKafkaControllerowns the broker (testcontainers,apache/kafka:3.8.0, KRaft), creates the twelve topics up front, and implements the hook exactly as #1738's javadoc sketches it:replyTimeoutMsruns out; a reply carrying someone else's id is skipped, one carrying none is taken and reported ascorrelationMatched = false;AsyncApiReplyDtoand judge nothing.kafka-clientsandtestcontainers-kafkaappear only in this test module, managed in the rootpom.xmlas the doc requires. Nothing is added to any shipped jar.The test
AsyncApiTestBasejoins the other bases ine2e-tests-utils:initAndRun, and helpers that readAsyncApiCallResultoff the solution.NcsKafkaEMTestruns 300 action evaluations and asserts:checkTriangle,bessjandremainderreached their result reply;expintandgammqreached both their result and their error — the two operations whose rejected inputs (x < 0;a ≤ 0 || x < 0) the schema does not rule out.bessj's order andremainder's operands are bounded in the schema, so their error replies need invalid data, which the search does not publish yet — worth knowing, and a natural next lever;NO_REPLY, noPUBLISH_FAILED.Because the driver is embedded, the SUT is instrumented, so the search covers lines and branches of a message-driven service as well: 662 targets in 26 seconds, identical on two runs.
Not here
The
createTests falseconstraint still stands, sorunTestHandlingFlakyis used rather than the compilation variant. Once the test writer exists, this is where its output gets compiled and run too.